Data Description:¶
The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.
Domain:¶
Object recognition
Context:¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information:¶
● All the features are geometric features extracted from the silhouette.
● All are numeric in nature.
Objective:¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
Importing the relevant libraries¶
In [1]:
import numpy as np # for dataframe handling
import pandas as pd #array handling
import seaborn as sns # plotting
sns.set(color_codes=True)
import matplotlib.pyplot as plt # plotting
%matplotlib inline
# For preprocessing the data
from sklearn import preprocessing
# To split the dataset into train and test datasets
from sklearn.model_selection import train_test_split
from scipy import stats
from sklearn import metrics
# To model the SVM
from sklearn import svm
#importing the Encoding library
from sklearn.preprocessing import LabelEncoder
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report,confusion_matrix
# To calculate the accuracy score of the model
from sklearn.metrics import accuracy_score
import warnings
warnings.filterwarnings('ignore')
#KFold cross validation
from sklearn.model_selection import KFold
#Build the model with the best hyper parameters
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import GridSearchCV
#Importing PCA for dimensionality reduction and visualization
from sklearn.decomposition import PCA
# Import Support Vector Classifier machine learning library
from sklearn.svm import SVC
In [2]:
# Reading the data as dataframe and print the first five rows
data = pd.read_csv('vehicle-1.csv')
data.head()
Out[2]:
In [3]:
data.columns
Out[3]:
In [4]:
data.shape #no of rows and columns in the dataframe
Out[4]:
There are 846 rows and 19 Columns in the DataFrame
In [5]:
data.dtypes # to get the data type of each attributes
Out[5]:
All the input features are numerics of type integer or float. Only the "class" is of type object, as it is categorical in nature.
Checking the presence of missing values¶
In [6]:
data.isnull().sum()
Out[6]:
There are missing Values in majority of the columns
Missing Values Treatment¶
In [7]:
# Split the dataset according to their class types
unique_vehicles = [data[data['class'] == veh] for veh in data['class'].unique()]
# Replaces the NULLs with the median of the respective feature
for unique_veh in unique_vehicles:
for col in unique_veh.columns[:-1]:
median = unique_veh[col].median()
unique_veh[col] = unique_veh[col].fillna(median)
# Join the splitted datasets back together and sort the index
data = pd.concat(unique_vehicles).sort_index()
In [8]:
data.isnull().sum()
Out[8]:
Descriptive statistics of each & every column¶
In [9]:
data.describe().transpose()
Out[9]:
Five Point Summary¶
In [10]:
summary=data.describe().T
summary[['min', '25%', '50%', '75%', 'max']]
Out[10]:
Univariate and Bivariate Analysis¶
In [11]:
data.skew(numeric_only = True)
Out[11]:
Skewness with positive values indicates data is skewed towards right. Skewness with negative values indicates data is skewed towards left
In [12]:
# A quick check to find columns that contain outliers
fig = plt.figure(figsize = (15, 7.2))
ax = sns.boxplot(data = data.iloc[:, 0:18], orient = 'h')
From above graph we found that many of the columns are having outliers
compactness¶
In [13]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['compactness'], showfliers=True,color='c').set_title("Distribution of 'compactness'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['compactness'],color='m').set_title("compactness Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['compactness'].plot.hist(color='g').set_title("compactness Vs Frequency");
Observation¶
- The column is almost Normally Distributed
- Mean and Median are almost the same
- Most of the values are between 87 to 100
- There are no outliers
circularity¶
In [14]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['circularity'], showfliers=True,color='c').set_title("Distribution of 'circularity'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['circularity'],color='m').set_title("circularity Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['circularity'].plot.hist(color='g').set_title("circularity Vs Frequency");
Observation¶
- The column is almost Normally Distributed
- Mean and Median are almost the same
- Most of the values are between 40 to 49
- There are no outliers
distance_circularity¶
In [15]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['distance_circularity'], showfliers=True,color='c').set_title("Distribution of 'distance_circularity'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['distance_circularity'],color='m').set_title("distance_circularity Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['distance_circularity'].plot.hist(color='g').set_title("distance_circularity Vs Frequency");
Observation¶
- The column is slightly skewed towards right.
- Mean is slightly greater than Median
- Most of the values are between 70 to 98
- There are no outliers
radius_ratio¶
In [16]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['radius_ratio'], showfliers=True,color='c').set_title("Distribution of 'radius_ratio'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['radius_ratio'],color='m').set_title("radius_ratio Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['radius_ratio'].plot.hist(color='g').set_title("radius_ratio Vs Frequency");
Observation¶
- The column is slightly skewed towards right.
- Mean is slightly greater than Median
- Most of the values are between 141 to 195
- There are outliers
pr.axis_aspect_ratio¶
In [17]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['pr.axis_aspect_ratio'], showfliers=True,color='c').set_title("Distribution of 'pr.axis_aspect_ratio'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['pr.axis_aspect_ratio'],color='m').set_title("pr.axis_aspect_ratio Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['pr.axis_aspect_ratio'].plot.hist(color='g').set_title("pr.axis_aspect_ratio Vs Frequency");
Observation¶
- The column is slightly skewed towards right.
- Mean is slightly greater than Median
- Most of the values are between 57 to 65
- There are outliers
max.length_aspect_ratio¶
In [18]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['max.length_aspect_ratio'], showfliers=True,color='c').set_title("Distribution of 'max.length_aspect_ratio'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['max.length_aspect_ratio'],color='m').set_title("max.length_aspect_ratio Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['max.length_aspect_ratio'].plot.hist(color='g').set_title("max.length_aspect_ratio Vs Frequency");
Observation¶
- The column is slightly skewed towards right.
- Mean is slightly greater than Median
- Most of the values are between 7 to 10
- There are outliers
scatter_ratio¶
In [19]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['scatter_ratio'], showfliers=True,color='c').set_title("Distribution of 'scatter_ratio'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['scatter_ratio'],color='m').set_title("scatter_ratio Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['scatter_ratio'].plot.hist(color='g').set_title("scatter_ratio Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 147 to 198
- There are no outliers
elongatedness¶
In [20]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['elongatedness'], showfliers=True,color='c').set_title("Distribution of 'elongatedness'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['elongatedness'],color='m').set_title("elongatedness Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['elongatedness'].plot.hist(color='g').set_title("elongatedness Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is Less than Median
- Most of the values are between 33 to 46
- There are no outliers
pr.axis_rectangularity¶
In [21]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['pr.axis_rectangularity'], showfliers=True,color='c').set_title("Distribution of 'pr.axis_rectangularity'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['pr.axis_rectangularity'],color='m').set_title("pr.axis_rectangularity Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['pr.axis_rectangularity'].plot.hist(color='g').set_title("pr.axis_rectangularity Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 19 to 23
- There are no outliers
max.length_rectangularity¶
In [22]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['max.length_rectangularity'], showfliers=True,color='c').set_title("Distribution of 'max.length_rectangularity'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['max.length_rectangularity'],color='m').set_title("max.length_rectangularity Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['max.length_rectangularity'].plot.hist(color='g').set_title("max.length_rectangularity Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 137 to 159
- There are no outliers
scaled_variance¶
In [23]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['scaled_variance'], showfliers=True,color='c').set_title("Distribution of 'scaled_variance'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['scaled_variance'],color='m').set_title("scaled_variance Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['scaled_variance'].plot.hist(color='g').set_title("scaled_variance Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 167 to 217
- There are outliers
scaled_variance.1¶
In [24]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['scaled_variance.1'], showfliers=True,color='c').set_title("Distribution of 'scaled_variance.1'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['scaled_variance.1'],color='m').set_title("scaled_variance.1 Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['scaled_variance.1'].plot.hist(color='g').set_title("scaled_variance.1 Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 318 to 587
- There are outliers
scaled_radius_of_gyration¶
In [25]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['scaled_radius_of_gyration'], showfliers=True,color='c').set_title("Distribution of 'scaled_radius_of_gyration'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['scaled_radius_of_gyration'],color='m').set_title("scaled_radius_of_gyration Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['scaled_radius_of_gyration'].plot.hist(color='g').set_title("scaled_radius_of_gyration Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 149 to 198
- There are no outliers
scaled_radius_of_gyration.1¶
In [26]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['scaled_radius_of_gyration.1'], showfliers=True,color='c').set_title("Distribution of 'scaled_radius_of_gyration.1'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['scaled_radius_of_gyration.1'],color='m').set_title("scaled_radius_of_gyration.1 Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['scaled_radius_of_gyration.1'].plot.hist(color='g').set_title("scaled_radius_of_gyration.1 Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 67 to 75
- There are outliers
skewness_about¶
In [27]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['skewness_about'], showfliers=True,color='c').set_title("Distribution of 'skewness_about'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['skewness_about'],color='m').set_title("skewness_about Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['skewness_about'].plot.hist(color='g').set_title("skewness_about Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 2 to 6
- There are outliers
skewness_about.1¶
In [28]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['skewness_about.1'], showfliers=True,color='c').set_title("Distribution of 'skewness_about.1'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['skewness_about.1'],color='m').set_title("skewness_about.1 Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['skewness_about.1'].plot.hist(color='g').set_title("skewness_about.1 Vs Frequency");
Observation¶
- The column is skewed towards right.
- Mean is greater than Median
- Most of the values are between 5 to 19
- There are outliers
skewness_about.2¶
In [29]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['skewness_about.2'], showfliers=True,color='c').set_title("Distribution of 'skewness_about.2'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['skewness_about.2'],color='m').set_title("skewness_about.2 Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['skewness_about.2'].plot.hist(color='g').set_title("skewness_about.2 Vs Frequency");
Observation¶
- The column is almost Normally Distributed
- Mean is slightly greater than Median
- Most of the values are between 184 to 193
- There are no outliers
hollows_ratio¶
In [30]:
plt.figure(figsize= (20,20))
#boxplot
plt.subplot(3,3,1)
sns.boxplot(data['hollows_ratio'], showfliers=True,color='c').set_title("Distribution of 'hollows_ratio'")
#distplot
plt.subplot(3,3,2)
sns.distplot(data['hollows_ratio'],color='m').set_title("hollows_ratio Vs Frequency")
#histogram plot
plt.subplot(3,3,3)
data['hollows_ratio'].plot.hist(color='g').set_title("hollows_ratio Vs Frequency");
Observation¶
- The column is skewed towrds left
- Mean is less than Median
- Most of the values are between 197 to 211
- There are no outliers
Outlier Treatement¶
In [31]:
# Plot the central tendency of the dataset
_, bp = data.boxplot(return_type='both', figsize=(20,10), rot='vertical')
fliers = [flier.get_ydata() for flier in bp["fliers"]]
boxes = [box.get_ydata() for box in bp["boxes"]]
caps = [cap.get_ydata() for cap in bp['caps']]
whiskers = [whiskers.get_ydata() for whiskers in bp["whiskers"]]
In [32]:
# Count the number of outlier data points present in each feature
for idx, col in enumerate(data.columns[:-1]):
print(col, '--', len(fliers[idx]))
There are 8 Columns in the dataset which contains outliers
In [33]:
# Treat the outlier data points
for idx, col in enumerate(data.columns[:-1]):
q1 = data[col].quantile(0.25)
q3 = data[col].quantile(0.75)
low = q1 - 1.5*(q3 - q1)
high = q3 + 1.5*(q3 - q1)
data.loc[(data[col] < low), col] = caps[idx * 2][0]
data.loc[(data[col] > high), col] = caps[idx * 2 + 1][0]
In [34]:
# Check the dataset after Outlier treatment
fig = plt.figure(figsize = (15, 7.2))
ax = sns.boxplot(data = data.iloc[:, 0:18], orient = 'h')
Target Column (Status) Distribution¶
In [35]:
class_counts = pd.DataFrame(data["class"].value_counts()).reset_index()
class_counts.columns =["Labels","class"]
class_counts
Out[35]:
In [36]:
# Check the frequency distribution of each target class
fig, axes = plt.subplots(1, 2, figsize=(16,6))
sns.countplot(data["class"], ax=axes[0], palette='rocket')
_ = axes[1].pie(data["class"].value_counts(), autopct='%1.1f%%', shadow=True, startangle=90, labels=data["class"].value_counts().index)
Observation¶
- From the above graph we understand that almost half of the data is car and percentage of van and bus is 23.5 and 25.8% respectively so it is in the ratio 2:1:1 and we also know there are two types of car , ie Saab 9000 and an Opel Manta 400.So the data is almost balanced
In [37]:
# Compare class wise mean
pd.pivot_table(data, index='class', aggfunc=['mean']).T
Out[37]:
Observations:¶
- There is not much of variation of any features in all the three vehicle types.
Influence of Different attributes on Class¶
In [38]:
for i in data:
if i != 'class' and i != 'name':
sns.catplot(x='class',y=i,kind='box',data=data)
Encoding the Target attribute¶
In [39]:
#Encoding of categorical variable
labelencoder_X=LabelEncoder()
data['class']=labelencoder_X.fit_transform(data['class'])
Pairplot¶
In [40]:
sns.pairplot(data,diag_kind='kde',hue='class');
In [41]:
plt.figure(figsize=(25, 25))
ax = sns.heatmap(data.corr(), vmax=.8, square=True, fmt='.2f', annot=True, linecolor='white', linewidths=0.01)
plt.title('Correlation of Attributes')
plt.show()
Based on Pair Plot and Correlation Matrix we Understand that¶
compactness is positively associated with circularity,distance_circularity,radius_ratio,scatter_ratio,pr.axis_rectangularity, max.axis_rectangularity, scaled_variance, scaled_variance.1 and is negatively associated with elonfatedness
circularity has positive association with distance_circularity, scatter_ratio, pr.axis_rectangularity, max.axis_rectangularity, scaled_variance, scaled_variance1, scaled_radius_of_gyration and has negative assiciation with elongatedness.
distance_circularity is positively associated with radius_ratio, scatter_ratio, pr.axis_rectangularity, max.axis_rectangularity, scaled_variance, scaled_variance1, scaled_radius_of_gyration and is negatively associated with elongatedness.
radius_ratio is positively associated with pr.axis_aspect_ratio, scatter_ratio, scaled_variance, scaled_variance1, scaled_radius_of_gyration is negatively associated with elongatedness.
pr.axis_aspect_ratio is positively associated with radius_ratio
skewness_about and skewness_about1 doesnot have much better correlation apart from themselves
scaled_radius_of_gyration.1 is negatively correlated with hollows_ratio and skewness_about.2.
elongatedness is negatively associated with almost all columns
Finding feature importance of data¶
In [42]:
#lets import extra tree classifier to find out the feature importance of data
from sklearn.ensemble import ExtraTreesClassifier
In [43]:
X=data.drop('class', axis=1)
y=data[['class']]
X.head()
Out[43]:
In [44]:
# Feature Importance plot using Random Forest Classifier
rf = RandomForestClassifier().fit(X, y)
pd.DataFrame(rf.feature_importances_, index = data.columns[:-1],
columns=['Importance']).sort_values('Importance').plot(kind='barh', figsize=(15,7), title='Feature Importance')
Out[44]:
Dropping of attributes with high correlation¶
In [45]:
X=X.drop(['scatter_ratio','pr.axis_rectangularity','scaled_variance.1', 'scaled_variance'], axis=1)
- Dropped scatter_ratio, pr.axis_rectangularity, scaled_variance.1, scaled_variance as they are having high correlations with many of the attributes, hence to reduce the burden of dimesions dropped those attributes.
- skewness_about and skewness_about1 doesnot have much better correlation and less importance value but after analyzing the pairplot is was visible that these are highly indepednent which will add as an advantage.
In [46]:
X.info()
- We droped few attributes to reduce multicoliniarity between attributes which may lead to curse of dimentionality and will impact the performance of models.
Split the data into train and test¶
In [47]:
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 10)
Performing Feature Scaling on the Data¶
In [48]:
from sklearn.preprocessing import StandardScaler
sc= StandardScaler()
X_train=sc.fit_transform(X_train)
X_test=sc.transform(X_test)
print(X_train.shape)
print(y_train.shape)
In [49]:
print(X_test.shape)
print(y_test.shape)
Support Vector Classifier¶
Hyperparameter Tuning¶
In [50]:
param_grid = [{'kernel': ['rbf'], 'gamma': [1,1e-1,1e-2, 1e-3, 1e-4, 1e-5],
'C': [0.01, 0.05,0.5,1, 10, 25, 50, 100]},
{'kernel': ['linear'], 'C': [0.001, 0.10, 0.1, 10]}]
grid = GridSearchCV(SVC(), param_grid, refit = True)
# fitting the model for grid search
grid.fit(X_train, y_train)
# print best parameter after tuning
print(grid.best_params_)
# print how our model looks after hyper-parameter tuning
print(grid.best_estimator_)
In [ ]:
In [51]:
# With the best hyper parameters found as given above, the final model can be built as given below
clf=SVC(kernel='rbf', C=25, gamma=0.01)
clf.fit(X_train, y_train)
prediction = clf.predict(X_test)
In [52]:
# check the accuracy on the training data
print('Accuracy on Training data: ',clf.score(X_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ', clf.score(X_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, prediction, average='macro'))
print("Classification Report:\n",metrics.classification_report(prediction,y_test))
Confusion matrix¶
In [53]:
cm=metrics.confusion_matrix(y_test, prediction, labels=[1, 0])
df_cm = pd.DataFrame(cm, index = [i for i in ["1","0"]],
columns = [i for i in ["Predict 1","Predict 0"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True, fmt="d")
Out[53]:
In [54]:
#Store the accuracy results for each kernel in a dataframe for final comparison
resultsDf = pd.DataFrame({'Model':['SVM - Hyper Parameter tuned(without PCA)'], 'Accuracy':clf.score(X_test , y_test) },index={'2'})
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[54]:
K-fold cross validation¶
In [55]:
#Scaling of independent attributes
from scipy.stats import zscore
XScaled = X.apply(zscore)
In [56]:
kfold = KFold(n_splits=50, random_state=7)
model = SVC()
results = cross_val_score(model, XScaled, y, cv=kfold)
In [57]:
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
In [58]:
#Store the accuracy results for each kernel in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Model':['SVM - KFold(without PCA)'], 'Accuracy': results.mean()},index={'3'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[58]:
Principal Component Analysis (PCA)¶
Standardize the data¶
In [59]:
XScaled=X.apply(zscore)
XScaled.head()
Out[59]:
Create a Covariance Matrix¶
In [60]:
# generating the covariance matrix and the eigen values for the PCA analysis
cov_matrix = np.cov(XScaled.T) # the relevanat covariance matrix
print('Covariance Matrix \n%s', cov_matrix)
Calculate Eigenvalues¶
In [61]:
#generating the eigen values and the eigen vectors
e_vals, e_vecs = np.linalg.eig(cov_matrix)
print('Eigenvectors \n%s' %e_vecs)
print('\nEigenvalues \n%s' %e_vals)
In [62]:
# the "cumulative variance explained" analysis
tot = sum(e_vals)
var_exp = [( i /tot ) * 100 for i in sorted(e_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp)
print("Cumulative Variance Explained", cum_var_exp)
In [63]:
# Plotting the variance expalained by the principal components and the cumulative variance explained.
plt.figure(figsize=(10 , 5))
plt.figure(figsize=(15,10))
plt.axhline(y=95, color='r', linestyle=':')
plt.bar(range(1, e_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, e_vals.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Observations:¶
- The above plot shows that, post PCA, employing 7 principal components we are able to explain more than 95% of the variance in the dataset.
- Hence we will be reducing our feature space to 7 dimensions without loosing much of information.
In [64]:
# Create a new matrix using the n components
X_projected = PCA(n_components=7).fit_transform(XScaled)
X_projected.shape
Out[64]:
In [65]:
#Converting PCA Transformed data from Array to Dataframe to visualise in the pairplot
pca_df=pd.DataFrame(X_projected)
In [66]:
sns.pairplot(pca_df, diag_kind = 'kde')
Out[66]:
Observation¶
- From above graphs we found that all the data points are in a cloud distribution
- There is no Relation between any components
In [67]:
# Divide the projected dataset into train and test split
X_projected_train, X_projected_test, y_train, y_test = train_test_split(X_projected, y, test_size=0.3, random_state=100)
X_projected_train.shape, X_projected_test.shape, y_train.shape, y_test.shape
Out[67]:
Support Vector Classifier After PCA¶
Hyperparameter tuning¶
In [68]:
param_grid = [{'kernel': ['rbf'], 'gamma': [1,1e-1,1e-2, 1e-3, 1e-4, 1e-5],
'C': [0.01, 0.05,0.5,1, 10, 25, 50, 100]},
{'kernel': ['linear'], 'C': [0.001, 0.10, 0.1, 10]}]
In [69]:
grid = GridSearchCV(SVC(), param_grid, refit = True)
# fitting the model for gridpca_y_trainsearch
grid.fit(X_projected_train,y_train)
Out[69]:
In [70]:
# print best parameter after tuning
print(grid.best_params_)
# print how our model looks after hyper-parameter tuning
print(grid.best_estimator_)
In [71]:
clf=SVC(kernel='rbf', C=10, gamma=0.1)
clf.fit(X_projected_train, y_train)
pca_prediction = clf.predict(X_projected_test)
In [72]:
# check the accuracy on the training data
print('Accuracy on Training data: ',clf.score(X_projected_train, y_train))
# check the accuracy on the testing data
print('Accuracy on Testing data: ', clf.score(X_projected_test , y_test))
#Calculate the recall value
print('Recall value: ',metrics.recall_score(y_test, pca_prediction, average='macro'))
#Calculate the precision value
print('Precision value: ',metrics.precision_score(y_test, pca_prediction, average='macro'))
print("Classification Report:\n",metrics.classification_report(pca_prediction,y_test))
In [73]:
cm=metrics.confusion_matrix(y_test, pca_prediction, labels=[1, 0])
df_cm = pd.DataFrame(cm, index = [i for i in ["1","0"]],
columns = [i for i in ["Predict 1","Predict 0"]])
plt.figure(figsize = (7,5))
sns.heatmap(df_cm, annot=True, fmt="d")
Out[73]:
In [74]:
tempResultsDf = pd.DataFrame({'Model':['SVM(PCA) - Hyper parameter tuned'], 'Accuracy': clf.score(X_projected_test, y_test)},index={'5'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[74]:
K-fold cross validation¶
In [75]:
kfold = KFold(n_splits=50, random_state=7)
model = SVC()
results1 = cross_val_score(model, X_projected, y, cv=kfold)
In [76]:
print("Accuracy: %.3f%% (%.3f%%)" % (results1.mean()*100.0, results1.std()*100.0))
In [77]:
tempResultsDf = pd.DataFrame({'Model':['SVM - KFold(PCA)'], 'Accuracy': results1.mean()},index={'6'})
resultsDf = pd.concat([resultsDf, tempResultsDf])
resultsDf = resultsDf[['Model','Accuracy']]
resultsDf
Out[77]:
Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components,¶
In [78]:
ax=sns.barplot(y="Model", x=("Accuracy"),data=resultsDf)
total = len(resultsDf["Accuracy"])
for p in ax.patches:
percentage = '{:.1f}%'.format(100 * p.get_width())
x = p.get_x() + p.get_width() + 0.02
y = p.get_y() + p.get_height()/2
ax.annotate(percentage, (x, y))
plt.show()
Conclusions¶
- From the plot between Explained variance ratio and Pricipal components we came know that taking 7 components can preserve more than 95% of variance of whole data.
- Using PCA we reduced the dimension from 18 to 7
When we applied Support vector classifier on the reduced Dimensions we got an accuracy of about 90.5% but the orginal dimensions scored better with 97% ### Effect of Principal Components Analysis can be more usefull in large datasets with more Dimensions
Multicolinearity and Curse of Dimensionality are the 2 major phenomenon which adversly impact any machine learning model. With higher degree of multicolinearity, model tend to leave behind the major information that is contained in the mathematical space of the input features. And with Curse of Dimensionality because of the feature space becoming increasingly sparse for an increasing number of dimensions of a fixed-size training dataset, model tend to overfit.
Principal Component Analyis helps adressing these problem and improves the model performance to a great extent.
In [ ]: